EARLIN: Early Out-of-Distribution Detection for Resource-Efficient Collaborative Inference

Collaborative inference enables resource-constrained edge devices to make inferences by uploading inputs (e.g., images) a server (i.e., cloud) where the heavy deep learning models run. While this setup works cost-effectively for successful inferences, it severely underperforms when model faces input samples on which was not trained (known as Out-of-Distribution (OOD) samples). If could, at least, detect that an sample is OOD, could potentially save communication and computation resources those workload. In paper, we propose novel lightweight OOD detection approach mines important features from shallow layers of pretrained CNN detects ID (In-Distribution) or based distance function defined reduced feature space. Our technique (a) without any retraining models, (b) does expose itself dataset (all parameters are obtained training dataset). To end, develop EARLIN (EARLy INference) takes partitions layer deploys considerably small part device rest cloud. By experimenting using real datasets prototype implementation, show our achieves better results than other approaches in terms overall accuracy cost tested against popular top benchmark datasets.